Large-scale environment mapping and immersive human-robot interaction for agricultural mobile robot teleoperation

Remote operation is a crucial solution to problems encountered in agricultural machinery operations. However, traditional video streaming control methods fall short in overcoming the challenges of single perspective views and the inability to obtain 3D information. In light of these issues, our research proposes a large-scale digital map reconstruction and immersive human-machine remote control framework for agricultural scenarios. In our methodology, a DJI unmanned aerial vehicle(UAV) was utilized for data collection, and a novel video segmentation approach based on feature points was introduced. To tackle texture richness variability, an enhanced Structure from Motion (SfM) using superpixel segmentation was implemented. This method integrates the open Multiple View Geometry (openMVG) framework along with Local Features from Transformers (LoFTR). The enhanced SfM results in a point cloud map, which is further processed through Multi-View Stereo (MVS) to generate a complete map model. For control, a closed-loop system utilizing TCP for VR control and positioning of agricultural machinery was introduced. Our system offers a fully visual-based immersive control method, where upon connection to the local area network, operators can utilize VR for immersive remote control. The proposed method enhances both the robustness and convenience of the reconstruction process, thereby significantly facilitating operators in acquiring more comprehensive on-site information and engaging in immersive remote control operations. The code is available at: https://github.com/LiuTao1126/Enhance-SF

Advanced Mapping of the Seafloor Using Sea Vehicle Mounted Sounding Technologies

A large proportion of the Earth’s surface is the deep sea. Numerous fields require access to seafloor topography and geomorphology. With the emergence of different types of underwater vehicles, especially the commercialization of near-seafloor micro-topographical mapping sonars, near-seafloor micro-topographical detection in the deep sea is possible. Near-seafloor micro-topographical exploration allows accurate detection of the seafloor using multibeam echosounder, side-scan sonar, and bathymetric side-scan sonar carried on-board various vehicles, including deep-tow, autonomous underwater vehicles, remotely operated vehicles, and human-occupied vehicles. Near-seafloor micro-topographical detection can obtain more accurate micro-topography and micro-geomorphology of the seafloor compared to full sea depth topographical detection. In this chapter, the basic principles of three types of near-seafloor micro-topographical mapping sonars are analyzed. Then, four types of underwater vehicles that are suitable for near-seafloor micro-topographical mapping are briefly discussed. Factors affecting mapping and detection results are presented using the Jiaolong human-occupied vehicle and its bathymetric side-scan sonar as an example. Next, the entire data processing and mapping methods are described. Finally, two typical detection results obtained by the Jiaolong bathymetric side-scan sonar in deep-sea are given

SuperChat: Dialogue Generation by Transfer Learning from Vision to Language using Two-dimensional Word Embedding and Pretrained ImageNet CNN Models

The recent work of Super Characters method using two-dimensional word embedding achieved state-of-the-art results in text classification tasks, showcasing the promise of this new approach. This paper borrows the idea of Super Characters method and two-dimensional embedding, and proposes a method of generating conversational response for open domain dialogues. The experimental results on a public dataset shows that the proposed SuperChat method generates high quality responses. An interactive demo is ready to show at the workshop.Comment: 5 pages, 2 figures, 1 table. Accepted by CVPR2019 Language and Vision Worksho